Method for selecting images in video of faces in the wild

ABSTRACT

Technologies are provided herein for selecting one or more subimages to be processed from a plurality of images received. A subset of images is identified, from the plurality of images received, that include a face image. The face image is assessed to generate assessment information regarding one or more of a size of the face image, a pose of the face in the face image, sharpness of the face image, and contrast quality of the face image. Each aspect of the assessment information is compared with corresponding criteria and, as a result of determining that the assessment information satisfies the criteria, facial recognition processing is performed on the face image.

BACKGROUND Technical Field

The present disclosure relates to systems and methods for processingfacial images.

Description of the Related Art

Some processor-based systems are configured to process images in orderto recognize objects, persons, places, animals, or other such featuresdepicted. Previously, users of some image processing systems had tomanually select which images to process because the processing power ofsuch systems is limited. However, in situations where there is a largeamount of image data, it becomes impracticable to process a substantialamount of data to recognize particular objects or persons. For instance,in public spaces, such as airports and city streets, the amount of humantraffic is far too large for any reasonable number of users to selectwhich images to process in an acceptable time period. Moreover, it maybe difficult and time consuming for a user to select which depictions ofpersons or objects in the images are suitable for processing. Thisproblem becomes even more challenging with video data that includes aseries of images as the users are required to process the video data inreal-time. Existing computer systems are insufficiently equipped orconfigured to solve this problem, and thus the ability of imageprocessing systems is significantly limited in such situations.

BRIEF SUMMARY

A computer-implemented method may be summarized as including receiving aplurality of images; and for each of one or more face images in theplurality of images: generating assessment information that includescontrast information regarding a contrast quality of the face image;comparing the assessment information for the face image with a set ofcriteria that includes a contrast quality criterion; determining, as aresult of comparing the assessment information with the set of criteria,whether the assessment information associated with the face imagesatisfies the set of criteria; and identifying, in response todetermining that the assessment information satisfies the set ofcriteria, the face image as being qualified for facial recognitionprocessing.

The assessment information may include sharpness information regarding asharpness of the face image, the set of criteria may include a sharpnesscriterion, and determining that the assessment information satisfies theset of criteria may include determining that the sharpness informationsatisfies the sharpness criterion. Generating the sharpness informationmay include performing a downsampling operation on the face image anddetermining a sharpness differential between the face image and a resultof the downsampling operation, and determining that the assessmentinformation satisfies the set of criteria may include determining thatthe sharpness differential satisfies the sharpness criterion. Generatingthe contrast information may include identifying a plurality of featuresin the face image that correspond to predetermined facial features;determining a plurality of regions of the face image that respectivelysurround a feature of the plurality of features; assigning contrastvalues to each region of the plurality of regions based on a maximumpixel contrast of pixels in each region; and determining a sum of thecontrast values of the plurality of regions. Determining that theassessment information satisfies the set of criteria may includedetermining that the sum of the contrast values satisfies the contrastcriterion. The set of criteria may include an orientation criterion, andgenerating the assessment information may include generating orientationinformation that includes estimating an orientation of the face in theface image, and determining that the assessment information satisfiesthe set of criteria may include determining that the orientationinformation satisfies the orientation criterion. The assessmentinformation may include size information identifying one or more sizedimensions of the face image, the set of criteria may include a sizecriterion, and determining that the assessment information satisfies theset of criteria may include determining that the size informationsatisfies a size threshold in the set of criteria.

The face image may be a first face image, the assessment information maybe first assessment information including first contrast information,and the method may further include obtaining a second face image;generating second assessment information of the second face image thatincludes second contrast information regarding a contrast quality of thesecond face image; determining that the second assessment informationsatisfies a second set of criteria that includes a second facialcontrast quality criterion; and generating, as a result of determiningthat the second assessment information satisfies the second set ofcriteria, a biometric signature of the second face image.

The computer-implemented method may further include generating a firstbiometric signature of the first face image and comparing the firstbiometric signature of the first face image to the second biometricsignature of the second face image. Each criterion of the second set ofcriteria may include different values than each corresponding criterionof the set of criteria.

The computer-implemented method may further include storing thebiometric signature as a result of determining that the secondassessment information represents an improvement over a stored biometricimage.

The second face image may be obtained at a second time and the methodmay further include storing a biometric signature of the first faceimage as being associated with an identity in data storage at a firsttime previous to the second time; and storing the biometric signature ofthe second face image as a result of a determination that a time periodbetween the first time and the second time exceeds a predeterminedthreshold.

The computer-implemented method may further include generating abiometric signature of the face image and a second biometric signaturefor a second face image consecutive to the face image in the pluralityof images; and determining an image differential based on a comparisonof the biometric signature and the second biometric signature, whereinanalyzing the face in the face image is performed as a result of theimage differential being less than a predetermined threshold for imageconsistency.

The method may further include detecting face images in at least asubset of images of the plurality of images, wherein generating theassessment information is in response to detecting the face image in thesubset of images.

The method may further include analyzing the face in the face image todetermine an identity of a person corresponding to the face.

A system, may be summarized as including one or more processors; andnon-transitory storage medium storing a set of instructions that, as aresult of execution by the one or more processors, cause the system to:receive a plurality of images; and for each of one or more face imagesin the plurality of images: generate assessment information thatincludes contrast information regarding a contrast quality of the faceimage; compare the assessment information with a set of criteria thatincludes a contrast quality criterion; and generate, as a result ofdetermining that the assessment information satisfies the set ofcriteria, result information indicating that the assessment informationassociated with the face image satisfies the set of criteria.

The instructions, as a result of execution by the one or moreprocessors, may further cause the system to transmit, to a secondsystem, the result information for authorizing the second system toperform facial recognition processing using the face image.

The instructions, as a result of execution by the one or moreprocessors, may further cause the system to generate a biometricsignature of the face image; and compare, as a result of determiningthat the assessment information satisfies the set of criteria, thebiometric signature with a plurality of biometric signatures stored indata storage.

The instructions, as a result of execution by the one or moreprocessors, may further cause the system to determine, as a result ofcomparing the biometric signature with the plurality of biometricsignatures, that the biometric signature does not correspond to a storedidentity; and create a new identity corresponding to the biometricsignature.

The instructions, as a result of execution by the one or moreprocessors, may further cause the system to determine, as a result ofcomparing the biometric signature with the plurality of biometricsignatures, that the biometric signature corresponds to a plurality ofstored identities; and merge the plurality of identities into a singlestored identity. The assessment information may include a size of theface image, pose information regarding an orientation of a face in theface image, and sharpness information regarding a sharpness of the faceimage, and the set of criteria may include a size criterion, anorientation criterion, and a sharpness criterion.

The instructions, as a result of execution by the one or moreprocessors, may further cause the system to identify a plurality offeatures in the face image that correspond to predetermined facialfeatures; determine a plurality of regions of the face image thatrespectively surround each feature of the plurality of features; computemaximum pixel difference values for the plurality of regions; andcompute an average pixel difference of the maximum pixel differencevalues, wherein the contrast information corresponds to the averagepixel difference.

A non-transitory computer-readable medium storing computer-executableinstructions that, as a result of execution by a processor, may besummarized as causing the processor to obtain a plurality of images; andfor each of one or more face images in the plurality of images: generateassessment information that includes contrast information regarding acontrast quality of the face image; compare the assessment informationwith a set of criteria that includes a contrast quality criterion; andgenerate, as a result of determining that the assessment informationsatisfies the set of criteria, result information indicating that theassessment information associated with the face image satisfies the setof criteria.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 shows a system for selecting, from a plurality of images, one ormore subimages to be processed according to one or more embodiments;

FIG. 2 shows an environment in which one or more images are evaluated todetermine whether to perform facial recognition processing;

FIG. 3 shows an environment in which image assessment information isevaluated;

FIG. 4 shows an environment in which the consistency of one or moreimages is verified;

FIG. 5 shows a diagram of a plurality of regions generated for a faceimage;

FIG. 6 shows a method for evaluating a plurality of images for facialrecognition processing;

FIG. 7 shows a method for generating sharpness information regarding theface image;

FIG. 8 shows a method for determining a plurality of regions in the faceimage;

FIG. 9 shows a method for generating contrast quality informationregarding the face image;

FIG. 10 shows a method for performing facial recognition processing;

FIG. 11 shows a method for updating contents of data storage involved infacial recognition processing;

FIG. 12 shows a method for verifying the consistency of images received;and

FIG. 13 shows a graphical representation representing a relationship ofcalculated values for contrast quality and normalized values forcontrast quality.

DETAILED DESCRIPTION

The following description, along with the accompanying drawings, setsforth certain specific details in order to provide a thoroughunderstanding of various disclosed embodiments. However, one skilled inthe relevant art will recognize that the disclosed embodiments may bepracticed in various combinations, without one or more of these specificdetails, or with other methods, components, devices, materials, etc. Inother instances, well-known structures or components that are associatedwith the environment of the present disclosure, including but notlimited to the communication systems and networks, have not been shownor described in order to avoid unnecessarily obscuring descriptions ofthe embodiments. Additionally, the various embodiments may be methods,systems, media, or devices. Accordingly, the various embodiments may beentirely hardware embodiments, entirely software embodiments, orembodiments combining software and hardware aspects.

Throughout the specification, claims, and drawings, the following termstake the meaning explicitly associated herein, unless the contextclearly dictates otherwise. The term “herein” refers to thespecification, claims, and drawings associated with the currentapplication. The phrases “in one embodiment,” “in another embodiment,”“in various embodiments,” “in some embodiments,” “in other embodiments,”and other variations thereof refer to one or more features, structures,functions, limitations, or characteristics of the present disclosure,and are not limited to the same or different embodiments unless thecontext clearly dictates otherwise. As used herein, the term “or” is aninclusive “or” operator, and is equivalent to the phrases “A or B, orboth” or “A or B or C, or any combination thereof,” and lists withadditional elements are similarly treated. The term “based on” is notexclusive and allows for being based on additional features, functions,aspects, or limitations not described, unless the context clearlydictates otherwise. In addition, throughout the specification, themeaning of “a,” “an,” and “the” include singular and plural references.

References to the term “set” (e.g., “a set of items”), as used herein,unless otherwise noted or contradicted by context, is to be construed asa nonempty collection comprising one or more members or instances.

References to the term “subset” (e.g., “a subset of the set of items”),as used herein, unless otherwise noted or contradicted by context, is tobe construed as a nonempty collection comprising one or more members orinstances of a set or plurality of members or instances. Moreover, theterm “subset,” as used herein, refers to a collection of one or moremembers or instances that are collectively smaller in number than theset or plurality of which the subset is comprised. For instance, asubset of a set of ten items will include less than ten items and atleast one item.

References to the term “module,” as used herein, is to be construed as acollection of hardware configured to perform a set of particularcomputing functions within a system. The hardware of a module mayinclude one or more processors that are specifically hardwired toperform one or more of the set of particular computing functions. Amodule may include a general purpose processor and memory storing a setof instructions that, as a result of execution by the general purposeprocessor, causes the module to perform one or more of the set ofparticular computing functions.

References to the term “subimage,” as used herein, refers to a portionof a image. A subimage, for example, may be a collection of pixels takenfrom an image that comprise fewer pixels than the number of pixels ofthe entire image.

FIG. 1 shows a system 100 for selecting, from a plurality of images, oneor more subimages to be processed according to one or more embodimentsherein. The system 100 includes one or more processors 102 that causethe system 100 to perform one or more operations described herein. Thesystem 100 receives a plurality of images 105 from a camera 106 andanalyzes the plurality of images 105 to determine which, if any, faceimages 114 to select for use in connection with facial recognitionoperations. A face image 114 may be an entire image 104 that depicts asingle face or may be a portion of an image 104 that depicts a singleface and that is smaller than the entire image 104. In some instances, aface image 114 refers to an image 104 or subimage thereof identified bythe system 100 as containing an image depicting a single human face.

The system 100 evaluates the plurality of images 105 and determineswhich, if any, of the face images contained in the plurality of images105 satisfy face image criteria 108. Face images determined assatisfying the face image criteria 108 are subjected to furtherprocessing, which may include processes related to identification of aperson shown in the face image. Because processes related toidentification of a person (e.g., facial recognition) are resourceintensive, determining face images that satisfy the face image criteria108 helps to ensure that those face images having certain qualitiescorrelated to a higher likelihood of identifying persons are processedfor facial recognition. As a result, face images having less desirablequalities for face recognition are discarded, or at least not used forfacial recognition, and the efficiency of resources directed to facialrecognition processes may be improved.

A face detection process 110 is performed on the plurality of images 105to identify a subset of images 112 (e.g., image 104) of the plurality ofimages 105 that include one or more face images. As part of or inconnection with the face detection process 110, the system 100 mayidentify face images 114 from the subset of images 112 and perform aface image assessment 116 on each of the face images 114. In particular,the system 100 assesses each of the face images 114 based on a set offactors and generates image assessment information 118 for each of theface images 114 based on a result of the assessment. The face image 114may be evaluated based on image characteristics as well as content ofthe face in the face image 114. The set of factors may include one ormore of a size of the face image 114, a pose of a face in the face image114, sharpness of the face image 114, and contrast quality of the faceimage 114.

The image assessment information 118 generated is evaluated based onface image criteria 108 to determine whether to advance the face image114 for facial recognition processing 122. The face image criteria 108includes a set of criteria corresponding to the aforementioned set offactors—namely, one or more of the size of the face image 114, the poseof the face in the face image 114, the sharpness of the face image 114,and the contrast quality of the face image 114. The set of criteria mayspecify a different threshold or condition corresponding to each of theset of factors. The system 100 compares 120 the information for eachfactor in the image assessment information 118 to the correspondingthreshold or condition in the face image criteria 108. As a result ofdetermining that the image assessment information 118 satisfies each ofthe set of criteria in the face image criteria 108, the system 100advances the face image 114 for facial recognition processing 122.Advancing the face image 114 for facial recognition processing 122 mayinclude storing the face image 114 in data storage for furtherprocessing, transmitting the face image 114 to another part of thesystem 100 or another system for facial recognition processing 122, orotherwise approving the face image 114 for facial recognition processing122. On the other hand, as a result of determining that the imageassessment information 118 fails to satisfy each of the set of criteria,the system 100 will not advance the face image 114 for facialrecognition processing 122 and may instead eliminate 124 the face image114 from consideration. Facial recognition processing 122 is describedin further detail below.

The system 100 may process each face image 114 identified in the subsetof images 112 to determine whether to perform facial recognitionprocessing 122 thereon. For each face image 114 approved for facialrecognition processing 122, the system 100 may store in persistentstorage the data corresponding to the face image 114 and may optionallystore the image assessment information 118 in association with the faceimage 114 data. Subsequent to evaluating all the face images 114identified in an image 104 of the subset of images 112, the system 100may discard the image 104 and obtain another image 104 of the subset ofimages 112 to evaluate. The plurality of images 105 correspond to aseries of images taken sequentially over time. The plurality of images105, for instance, may be a stream of video data, a video data object,or a collection of image data objects sequentially captured by thecamera 106.

The system 100 may continue to obtain images 104 (e.g., from the camera106, from another system) until a suitable face image 114 is identifiedfor facial recognition processing 122 or until no more images 104 remainfor processing. The system 100 may replace stored face image 114 with aface image 114 determined as having superior or better characteristics(e.g., sharpness, pose) than the stored face image 114. In someinstances, the biometric signature of a face image 114 determined ashaving superior characteristics may be stored in place of or in additionto the stored biometric signature of the face image 114 having lessdesirable characteristics. As a result, the system 100 is configured toobtain better images and biometric signatures for comparison, therebyimproving the accuracy of the system 100 over time. Further descriptionof the process of obtaining better samples for comparison is describedbelow with respect to FIG. 11.

The camera 106, from which the plurality of images 105 is obtained, iselectrically communicatively coupled to the system 100 and provides theplurality of images 105 as an input to the system 100. The camera 106may be directly coupled to the system 100 via a wired connection or maybe remotely coupled to provide the plurality of images 105 to the system100, for example, over a wired or wireless communication network. Insome embodiments, a plurality of cameras 106 coupled to the system 100may each provide a plurality of images 105 to the system 100. Althoughthe plurality of images 105 are shown as being received from the camera106, the plurality of images 105 may be received from a source otherthan the camera 106. For instance, the plurality of images 105 may bereceived over a network (e.g., local area network, wide area network)and/or via an intermediate device, such as a network router or a server.In some instances, the plurality of images 105 may be stored in memoryat a first time and provided to the system 100 at a second time laterthan the first time.

In some embodiments, the one or more processors 102 are electricallycommunicatively coupled to memory 126 that stores a set of instructions128 that, as a result of execution by the one or more processors 102,cause the system 100 to perform one or more operations described herein.In some embodiments, the one or more processors 102 may be a devicehaving hardware specifically configured to perform at least some of theoperations described herein. For instance, the one or more processors102 may include application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), system-on-a-chip (SoC), or otherspecialized or customizable computing hardware hardwired to perform atleast some of the operations described herein.

FIG. 2 shows an environment 200 in which one or more images 104 areevaluated by the system 100 to determine whether to perform facialrecognition processing according to one or more embodiments describedherein. The environment 200 includes a face detection module 202 thatdetects images of faces, also referred to as face images, in each of theplurality of images 105. The face detection module 202 receives theplurality of images 105 provided by the camera 106, processes each image104 of the plurality of images 105, and identifies images 104 containingone or more face images 114 of persons. The face detection module 202implements one or more algorithms for detecting facial features in theimages 104. The face detection module 202 may, for instance, detectimages of all or part of faces, such as images taken from the side thatdo not include both eyes of the subject. The face detection module 202may detect the presence of a face in an image based on detecting a matchbetween a group of pixels therein and attributes indicated ascorresponding to human faces. The face detection module 202 may identifyregions of pixels in each image 104 corresponding to a face.

The face detection module 202 may generate and provide output to animage assessment module 204 indicating that a face was detected in animage 104 received. The face detection module 202 may output the subsetof images 112 containing at least one face image 114 to the imageassessment module 204. In some embodiments, the face detection module202 may indicate, to the image assessment module 204, which images 104were detected as containing a face to the image assessment module 204and information indicating one or more regions of pixels in the image104 detected as containing the face. The image assessment module 204 maythen extract each of the face images 114 from the regions of pixelsidentified for assessment. In some embodiments, the face detectionmodule 202 may extract the groups of pixels detected as the face image114, and send the extracted face images 114 to the image assessmentmodule 204.

The image assessment module 204 receives or otherwise obtains the faceimages 114 indicated by the face detection module 202 and assesses theface images 114. For each face image 114, the image assessment module204 generates image assessment information 118 regarding one or more ofthe set of factors discussed herein—the size of the face image 114, thepose of the face in the face image 114, the sharpness of the face image114, and the contrast quality of the face image 114. The imageassessment module 204 evaluates the image assessment information 118using the face image criteria 108 to determine whether the face image114 has attributes that justify subjecting the face image 114 to facialrecognition processing 122.

If the image assessment module 204 determines that the image assessmentinformation 118 for a particular face image 114 satisfies the face imagecriteria 118, the image assessment module 204 may generate an assessmentoutput 206 that approves the face image 114 for facial recognitionprocessing 122. The output 206 may include the face image 114 or mayinclude information indicating a location (e.g., in memory, networklocation, database location) from which the face image 114 may beobtained. The output 206 may include result information 208 indicatingvarious information related to the assessment performed. For instance,the result information 208 may include some or all of the imageassessment information 118. The result information 208 may indicateinformation indicating a margin of satisfaction of the image assessmentinformation 118 for each factor with respect to the face image criteria108—for instance, numerical values indicating how much the imageassessment information 118 exceeded each factor of the face imagecriteria 108. In some embodiments, the result information 208 mayindicate one or more operations to be performed in the facialrecognition processing 122 as a result of the assessment by the imageassessment module 204. If the image assessment module 204 determinesthat the image assessment information 118 does not satisfy the faceimage criteria 108, the image assessment module 204 may discard the faceimage 114 and obtain another face image 114 for assessment. The imageassessment module 204 may not generate or provide the output 206 for aface image 114 as a result of determining that the corresponding imageassessment information 118 fails to satisfy the face image criteria 108.

As a result of receiving the output 206 from the image assessment module204, an image processing module 210 may include performing facialrecognition processing on the face image 114 in accordance with theoutput 206. Performing facial recognition processing by the imageprocessing module 210 may include generating a biometric signature 212of the face image 114, which can be used to determine an identity of theperson depicted in the face image 114. In some embodiments, thebiometric signature 212 is a face template having features correspondingto features of a subject's face. In some embodiments, the imageprocessing module 210 may be part of a second system that is separatefrom the system 100. The second system, for example, may be separatedfrom and connected to the system 100 by a network 218, such as a localarea network or a wide area network.

The image processing module 210 may perform various operations involvingthe biometric signature 212 depending on the result info 208, which mayinclude at least some of the image assessment information 118. Forinstance, the image processing module 210 may perform operations todetermine an identity of the person in the face image 114 as a result ofthe image assessment information 118 satisfying a first set ofthresholds specified in the face image criteria 108. The operations todetermine the subject's identity may include comparing the biometricsignature 212 with various previously-obtained stored biometricsignatures 216 stored in data storage 214. The stored biometricsignatures 216 may have a corresponding identity of a person associatedtherewith in the data storage 214. In some embodiments, an identity of aperson may be a name or alphanumeric combination (e.g., username,employee identification number) associated with a person. An identity ofa person may be information other than a name or alphanumericcombination in some embodiments. For example, an identity of a personmay be information associated with a previous instance in which a faceimage 114 of a particular person was captured, such as informationregarding time, date, location, etc. The system 100 may generate aunique alphanumeric value for the identity of the person and store thealphanumeric value in data storage 214.

As a result of determining that the biometric signature 212 issufficiently similar to a stored biometric signature 216, the imageprocessing module 210 may generate an output indicating the identityassociated with the stored biometric signature 216. The image assessmentinformation 118 for each of the stored biometric signatures 216 may bestored in the data storage 214 in association with the correspondingstored biometric signature 216. Moreover, each of the stored biometricsignatures 216 may be associated with information indicating an identityof the person pictured in the corresponding face image 114 (if known)and may include information indicating a time and date on which thecorresponding face image 114 was captured or stored.

The image processing module 210 may perform operations to update, in thedata storage 214, the stored biometric signature 216 depending on theresult info 208. In response to determining that the image assessmentinformation 118 satisfies a second set of thresholds specified in theface image criteria 108, the image processing module 210 may cause thebiometric signature 212 to be stored in the data storage 214. The imageprocessing module 210 may associate the biometric signature 212 nowstored in the data storage 214 with a subject's identity. A name orother alphanumeric identifier for a person may be associated with thebiometric signature 212. In some instances, another one of the storedbiometric signatures 216 may be disassociated with the identity suchthat the biometric signature 212 replaces an older stored biometricsignature 216. In some instances, the biometric signature 212 may bemerged with an identity such that a plurality of biometric signatures216 are stored in the data storage 214 in association with a singleidentity.

The biometric signature 212 generated may include an array of valuesrepresentative of a comparison between the face image 114 and acorresponding template stored in the memory 126. The biometric signature212 may, for instance, represent differences in Euclidean distancebetween points in the face image 114 and points in the blank template.The stored template may include a multidimensional vector in Euclideanspace. The memory 126 may store a plurality of templates that correspondto difference poses for face images 114 obtained. The instructions 128(FIG. 1) may include instructions that cause the one or more processors102 to implement an artificial intelligence model to generate thetemplate. The artificial intelligence model may include deep neuralnetwork (DNN) models and principles in which multilayered neuralnetworks learn to generate templates and/or biometric signatures basedon stored data regarding templates and face images. The system 100implements DNN technologies to identify relevant features in face imagesand generate corresponding templates and biometric signatures. Thestored biometric signatures 216 may be generated in the same way as thebiometric signature 212.

In some embodiments, the image assessment module 204 and the imageprocessing module 210 may be part of the same system 100 and even partof the same device in some implementations. In some embodiments, theimage assessment module 204 may send the output 206 over thecommunication network 218 to the image processing module 210. Thenetwork 218 may be a wide area network (e.g., internet) in which theimage processing module 210 is remotely located to the image assessmentmodule 204. In some embodiments, the network 218 may be a local areanetwork.

Facial recognition can be a computationally intensive process; however,determining which face images 114 to process has been a complex anddifficult challenge. The image processing module 210 performs facialrecognition processing on select face images 114 that satisfy the faceimage criteria 118. This process of evaluating the face images 114 basedon the set of factors identified herein helps to ensure that select faceimages 114 having characteristics suited to successfully recognize theidentity of persons imaged are processed. Therefore, the imageprocessing module 210 performs facial recognition processing on faceimages 114 most likely to achieve a worthwhile result, thereby improvingthe computational efficiency of the system 100.

FIG. 3 shows an environment 300 in which image assessment information118 is evaluated based on the face image criteria 108 according to oneor more embodiments. The environment 300 shows operations performed bythe image assessment module 204 in connection with the system 100. Imageassessment information 118 is generated based on a face image 114received from the face detection module 202. The image assessmentinformation 118 includes a set of alphanumeric values 302 representativeof various characteristics of the face image 114. The image assessmentinformation 118 may be formatted as an array, such as a one-dimensionalarray in which the alphanumeric entries are separated by delimiters(e.g., semi-colon, comma, tab). The image assessment information 118 maybe formatted as a multi-dimensional array or table having multiplecolumns and rows.

The image assessment information 118 includes size information 304representative of a size of the face image 114. The size information 304may specify a number of pixels contained in the face image 114, thewidth of the face image 114 (e.g., in pixels), a height of the faceimage 114 (e.g., in pixels), or an amount of data storage occupied bythe face image. For example, the size information 304 in the environment300 indicates that the face image 114 has 50176 pixels.

The image assessment information 118 includes pose information 306representative of a pose of a person's face or head in the face image114. The pose information 306 may indicate an orientation of the facerelative to an optical axis of the camera 106 capturing the face image114. The image assessment module 204 may generate the pose information306 based on the features identified in the person's face and head inthe face image 114. For instance, if the top of a person's head and aright eye of the subject are visible in the face image 114, the poseinformation 306 may indicate that the head is oriented downward and tothe left of the optical axis of the camera 106. The pose information 306may have a format indicating rotation about one or more axes. The poseinformation 306 shown, for example, indicates that the head in the faceimage 114 is rotated 10° about a vertical axis and 0° about a horizontalaxis. In some embodiments, the pose information 306 may includealphanumeric values representative of various features that can beidentified in the face image 114, which may correspond to rotationalstates of the head shown in the face image 114.

The image assessment information 118 also includes sharpness information308 representative of the sharpness of the face image 114. Sharpnessreflects a rate or quickness of change between boundaries of color ortone in the face image 114. The image assessment module 204 may generatethe sharpness information 118 based on an operation involving adecimation operation, as described below. The sharpness information 308may include an alphanumeric value indicating a sharpness level of theface image 114.

The image assessment information 118 also includes contrast qualityinformation 310. Contrast quality is an assessment of the overallquality of contrast for the face image 114. The image assessment module204 may generate the contrast quality information 310 based onevaluations of contrast in certain areas of a face in the face image114. The contrast quality information 310 may include one or morealphanumeric values indicating the contrast quality of the face image114.

The image assessment module 204 determines whether to advance the faceimage 114 for facial recognition processing as a result of thecomparison 120 between the image assessment information 118 and the faceimage criteria 108. The face image criteria 108 may include one or moreof size criteria 312, pose criteria 314, sharpness criteria 316, andcontrast quality criteria 318. The face image criteria 108 may beformatted as a one-dimensional de-limiter separated array, or amulti-dimensional array, as described above with respect to the imageassessment information 118. The face image criteria 108 may includeBoolean logic, conditions, and/or relationships regarding correspondinginformation in the image assessment information 118.

In the environment 300, the size criteria 312 specify one or moreacceptable size ranges or thresholds for the size information 304 of theface image 114. The size criteria 312, for example, may specify aminimum width for the face image 114, a minimum height for the faceimage 114, and/or a minimum total pixel count for the face image 114.The pose criteria 314 specify acceptable poses for the face imaged inthe face image 114, as indicated by the pose information 306. By way ofnon-limiting example, the pose criteria 314 may include acceptableranges for orientation of the head in the face image 114 or acceptablegroups of facial features represented in the face image 114. Thesharpness criteria 316 specify one or more acceptable sharpness rangesor thresholds for the sharpness information 308 of the face image 114.The contrast quality criteria 318 specify one or more acceptablecontrast quality ranges or thresholds for the contrast quality 310 ofthe face image 114. In some embodiments, some of the face image criteria108 may be dependent upon the image assessment information 118. Forexample, if the pose information 306 is within a first set of ranges,the face image criteria 108 may specify corresponding set of sizecriteria 312, sharpness criteria 316, and/or contrast quality criteria318.

The image assessment module 204 performs comparisons between the values302 for the image assessment information 118 and corresponding criteriain the face image criteria 108. As a result of determining that theimage assessment information 118 satisfies the face image criteria 108,the image assessment module 204 may generate the output 206 advancingthe face image 114 for facial recognition processing. The output 206 mayinclude an indication that the face image 114 is satisfactory and mayinclude the face image 114 or a reference to a location where the faceimage 114 may be obtained. The output 206 may include the assessmentresult information 208, which may include at least some of the imageassessment information 118 or information related thereto. As a resultof receiving the output 206, the image processing module 210 may performvarious facial recognition processing 122 using the face image 114.

The output 206 generated may be based at least in part on the face imagecriteria 108, which may include information, such as a classification,to be generated based on the values 302 of the image assessmentinformation 118. The face image criteria 108 may specify that theassessment output 206 should include a classifier corresponding to anoperation to be performed if the values 302 fall within certain ranges.For instance, as a result of the values 302 falling within a first rangespecified in the face image criteria 108, the image assessment module204 would generate an output 206 that includes first information thatwould cause the image processing module 210 to attempt to recognize theface in the face image 114. As a result of the values 302 falling withina second range specified in the face image criteria 108, the imageassessment module 204 would generate an output 206 that includes secondinformation that would cause the image processing module 210 to attemptto store the biometric signature 212 for the face image 114 as a newstored biometric signature 216 for future facial recognition.

FIG. 4 shows an environment 400 in which the system 100 verifiesconsistency of the images 104 obtained according to one or moreembodiments. The system 100 evaluates the images 104 obtained to helpensure that an inconsistent or unrepresentative image 104 is not usedfor facial recognition purposes. For instance, if there is a significantamount of motion during the image capture or there is cigarette smokepassing through the image, these occurrences may result in a biometricsignature 212 that is not adequately representative of the subject.Therefore, using the biometric signature 212 in such a situation couldlead to inaccurate facial recognition results and waste resources. Thesystem 100 verifies consistency of images obtained before proceeding toperform facial recognition processing 122. Aspects of the environment400 may be performed by the image assessment module 204 and/or the imageprocessing module 210.

The environment 400 involves obtaining a plurality of images 105sequentially captured by the camera 106 over time, as described hereinwith respect to FIG. 1. The camera 106 may capture at least a firstimage 104 a and a second image 104 b sequential to the first image 104 afor consistency verification. The plurality of images 105 obtained arenot necessarily immediately sequential—for example, a time period (e.g.,10 milliseconds, 1 second, 2 seconds) may separate first time at whichthe first image 104 a is captured and a second time at which the secondimage 104 b is captured. In some instances, there may be one or moreimages captured between the first image 104 a and the second image 104b.

One of the images 104 a and 104 b may contain the face image 114 thatthe system 100 is evaluating for facial recognition processing 122.Although only two images 104 a and 104 b are depicted and described, thesystem 100 may evaluate more than two images 104 in connection withimage consistency verification. In some embodiments, the signatures 402obtained may be of particular portions or subimages of the images 104.For instance, the signature 402 a may be of a portion of the image 104 adetected as being a face image 114 and the signature 402 b may be of acorresponding portion of the image 104 b. In some embodiments, thesignature 402 a may be the biometric signature 212 of the face image 114of a face captured at a first time, and the signature 402 b may be abiometric signature of the face image 114 of the face captured at asecond time after the first time. Signatures 402 are generated that eachcorresponds to one of the images 104. Generation of the signatures 402 aand 402 b may be performed according to the process discussed forgenerating biometric signatures 212 herein.

The system 100 may perform a comparison 404 between the first signature402 a and the second signature 402 b to determine consistency betweenthe signatures. The signatures 402 a and 402 b may be represented asrespective arrays of values corresponding to multidimensional vectors inEuclidean space. The system 100 may determine differentials betweencorresponding values in arrays to the signatures 402 a and 402 b todetermine how consistent the images 104 a and 104 b are with each other.The system 100 may compare 404 the differentials with one or morethresholds or ranges and determine that the images 104 a and 104 b arewithin an acceptable range of consistency. If the signatures 402 a and402 b are determined to be sufficiently similar or close in vectorspace, an output 408 is generated indicating that the images 104 a and104 b are verified as consistent. If the signatures 402 a and 402 b areinsufficiently similar, the system 100 may not generate the output 408or generate an output disapproving facial recognition processing 122 forthe face image 114. In some embodiments, the system 100 may considermore than two sequential images 104 or subimages in the consistencyverification of the environment 400. For example, the system 100 mayverify that three signatures 402 corresponding to three sequentialimages 104 are sufficiently similar as a condition precedent to issuingthe output 408.

In some embodiments, the system 100 may perform the facial recognitionprocessing 122 as a result of generating both the output 408 verifyingimage consistency and the output 206 verifying that the image assessmentinformation 118 satisfies the face image criteria 108 set of performingfacial recognition processing 122. Once the consistency of the images104 and the quality of the face image 114 are verified, the system 100may perform the facial recognition processing 122 in accordance with theoutput 206, as discussed herein. The system 100 may, for example,determine an identity of the person depicted in the face image 114 orstore the face image 114 in the data storage 214 as one of the storedbiometric signatures 216. The efficiency and reliability of the system100 for performing facial recognition processing 122 may be improved byprocessing certain face images 114 of sufficiently high quality andconsistency.

FIG. 5 shows a diagram 500 of a plurality of regions of the face image114 according to one or more embodiments. The diagram 500 is generatedby the system in connection with generating the contrast qualityinformation 310. The system 100 identifies a plurality of particularfeatures of the face in the face image 114 and determines a plurality ofregions in the face image 114 respectively centered about the particularfeatures of the face in the face image 114. The system 100 may generateor identify, for each region, a vertical axis 501 and a horizontal axis503 that are each centered within a region.

The system 100 identifies pupil centers of the left and right eyes inthe face image 114 and determines a first region 502 centered on a leftpupil 505 and a second region 504 centered on a right pupil 507. Thesystem 100 also identifies a left mouth corner 509 and a right mouthcorner 511 in the face image 114 and determines a third region 506centered around the left mouth corner 509 and a fourth region 508centered on the right mouth corner 511. The system 100 may alsodetermine a fifth region 510 between the first region 502 and the secondregion 504 and a sixth region 512 between the third region 506 and thefourth region 508. The system 100 may further determine a seventh region514 between the fifth region 510 and the sixth region 512. The system100 may also determine a first horizontal line 516 extending between thevertical axis 501 centered on the left pupil and the vertical axis 501centered on the right pupil. The system 100 may also determine a secondhorizontal line 518 extending between the vertical axis 501 centered onthe left mouth corner 507 and the vertical axis 501 centered on theright mouth corner 509. In some instances, features in the face image114, such as the left and right pupils or the left and right mouthcorners, may not be horizontally aligned (i.e., located at differentvertical positions). In such instances, the first horizontal line 516 orthe second horizontal line 518 may be positioned halfway between theoffset facial features.

The sizes of each of the plurality of regions may be based on distancesbetween the particular features of the face image 114 identified. Awidth of at least some of the regions may be based on a first distanced1 between the left pupil center and the right pupil center. The firstdistance d1 may be a distance in a horizontal direction parallel tohorizontal axis of the face image 114. A width of at least some of theregions may be based on a second distance d2 between the left mouthcorner and the right mouth corner. The second distance d2 may be adistance in a horizontal direction parallel to the horizontal axis ofthe face image 114. A height of at least some of the regions may bebased on a third distance d3 between the mouth and the eyes. Thedistance d3 may be a distance between a center horizontal axis of thefifth region 510 and a center horizontal axis of the sixth region 512.

In some instances, there may be more or fewer than seven regions. Forinstance, if a single eye is not visible in the face image 114, thesystem 100 may decide not to determine the region corresponding to theother non-visible eye. As a result, the sizes of the regions may bedetermined in a different manner. If the right eye is not visible, forexample, a horizontal size of the first region 502 may be determined asthe distance between the center of the left pupil to a bridge portion ofthe nose in the face image 114. As a result of determining the pluralityof regions, the system 100 may use the plurality of regions to generatethe contrast quality information 310.

FIG. 6 shows a method 600 for evaluating a plurality of images forfacial recognition processing according to one or more embodiments. Themethod 600 is performed by one or more components of the system 100,such as the image assessment module 204 and the face detection module202. The method 600 begins by receiving 602 a plurality of images 105.The plurality of images 105 are generated by the camera 106 and may bereceived therefrom or over a network. Next, the system 100 detects 604faces in one or more of the images 104. For example, the system 100 maydetect, in the one or more images, a set of facial features qualifyingas a face image 114. Detecting 604 the faces may include, for each image104 detected as including a face image 114, identifying a region in theimage 104 containing the face and generating information useable toprocess the face image 114. The system 100 may identify a set ofcoordinates corresponding to the region containing the face image 114,or extract information representative of the face image 114 for furtherprocessing. If no faces are detected in the plurality of images 105, thesystem 100 may wait to receive 602 another plurality of images 105.

The system 100 then proceeds to generate 606 assessment information 118for each face image 114 detected in the plurality of images 105.Generating 606 the assessment information 118 includes generating thesize information 304, the pose information 306, the sharpnessinformation 308, and the contrast quality information 310, as describedherein.

The method 600, in some embodiments, may include determining 607 whetherthe images 104 received are consistent. For instance, the system 100 maygenerate a biometric signature 212 of the face image 114 in a firstimage 104 of the plurality of images 105 received and generate abiometric signature 212 of the face image 114 in a second image 104 thatis sequential to the first image 104 in the plurality of images 105. Thesystem 100 may compare the first and second biometric signatures 212 todetermine whether they are sufficiently similar. If so, the method 600may proceed; otherwise, the method 600 may return to a previousoperation, such as by obtaining 612 a next face image 114. In someembodiments, the system 100 may determine consistency of a first image104 and a second image 104 sequential to the first image 104 in theplurality of images 105 received prior to generating 606 assessmentinformation 118 for face images 114 detected. In some embodiments,determining 607 whether the face images 114 are consistent may not beperformed. Further description of the determination 607 is describedbelow with respect to FIG. 12 and elsewhere herein.

Thereafter, the assessment information 118 is compared with the faceimage criteria 108 and the system 100 determines 608 whether theassessment information 118 satisfies the face image criteria 108.Determining 608 whether the assessment information 118 satisfies theface image criteria 108 may include one or more of a comparison of thesize information 304 with the size criteria 312, a comparison of thepose information 306 with the pose criteria 314, a comparison of thesharpness information 308 with the sharpness criteria 316, and acomparison of the contrast quality information 310 with the contrastquality criteria 318.

If, on the other hand, it is determined that the assessment information118 does not satisfy the face image criteria 108, the method 600proceeds by obtaining 612 the next face image 114 for evaluation. If nomore face images 114 remain for processing, the system 100 waits toreceive further images 104 for evaluation.

As a result of determining 608 that the assessment information 118satisfies the face image criteria 108, the method proceeds by advancing610 the face image 114 corresponding to the assessment information 118for facial recognition processing. By determining 608 whether theassessment information 118 satisfies the face image criteria 108 beforeperforming facial recognition processing, the system 100 obviatesunnecessary expenditure of potentially expensive computationalresources, storage, or network bandwidth on face images 114 that haveundesirable characteristics for facial recognition processing. Advancing610 the face image 114 for facial recognition processing causes thesystem 100 to compare the biometric signature 212 of the face image 114with stored biometric signatures 216 in the data storage 214. Forinstance, facial recognition processing may include determining whetherthe biometric signature 212 is a match for a stored biometric signatureor whether the biometric signature 212 corresponds to a new identitythat the system 100 should learn. Further description of facialrecognition processing is described herein with respect to FIG. 11 andelsewhere herein. The system 100 may return information indicating anidentity of a person depicted in the face image 114 evaluated as aresult of the face image processing.

FIG. 7 shows a method 700 involved in generating the sharpnessinformation 308 according to one or more embodiments. The method 700 maybe performed by the image assessment module 204 of the system 100. Themethod 700 is performed in connection with generating 606 the assessmentinformation 118 of the method 600 and as described elsewhere herein. Themethod 700 begins by obtaining a face image 114 from an image 104, whichmay involve extracting or copying information representative of a regiondetected as including a face in the image 104. The face image 114obtained 702 is stored in memory of the system 100 (e.g., random-accessmemory) for sharpness evaluation.

Next, the method 700 includes determining 704 edge information of theface image 114 obtained. Edge information indicates a sharpness oftransition at boundaries between different colors, tones, etc., in theface image 114. The edge information may include an array containingnumerical values representative of differences in color, tone, etc.,between adjacent or nearby pixels of the face image 114. The array maybe a one-dimensional array or multi-dimensional array including entriesseparated by delimiters (e.g., semi-colon, tab, comma). Higher numericalvalues may indicate a sharper transition in characteristics of adjacentor nearby pixels whereas a lower numerical value may indicate arelatively less sharp transition. The edge information may be generated,for example, by processes including performing a Fourier transform(e.g., Fast Fourier Transform, Discrete Fourier Transform) on the faceimage, application of a Sobel filter, or any such appropriate an edgedetection algorithm.

The method 700 includes downsampling 706 the face image 114 to generatea downsampled version of the face image 114. Downsampling 706 mayinvolve taking a first pixel from the face image 114 and then every Nthpixel (where N is an integer value) thereafter until there are no morepixels remaining to sample in the face image 114. One example ofdownsampling 706 is by performing a decimation operation in which thesample rate of the face image 114 is reduced by an integer factor.Downsampling 706 may also include application of a finite impulseresponse filter or Chebyshev filter on the face image 114. As a resultof downsampling 706 the face image 114, a downsampled face image isobtained. At 708, the method 700 includes determining edge informationfor the downsampled face image in a manner similar to determining 704the edge information of the face image 114.

Next, the method 700 includes determining 710 a sharpness differentialbetween edge information for the face image 114 and the edge informationfor the decimated face image. Determining 710 the sharpness differentialmay include determining differences between values in the edgeinformation for the face image 114 and corresponding values in thedownsampled face image. The system 100 may determine a maximum oraverage value for a portion of the edge information corresponding to aparticular subregion of the face image 114, and compare that value witha portion of edge information of a corresponding subregion of thedecimated face image. The system 100 may compute edge informationdifferences for subregions of the entire face image 114 and generatesharpness information corresponding to the differences in the edgeinformation. Larger values for the sharpness information 308 mayindicate that the face image 114 has relatively high sharpness, which isfavorable for proper evaluation of a biometric signature 212 of the faceimage 114. Conversely, lower values for the sharpness information 308may indicate that the face image 114 has relatively lower sharpness,which is unfavorable for proper evaluation of the face image 114.

FIG. 8 shows a method 800 involved in generating contrast qualityinformation 310 according to one or more embodiments. The method 800 maybe performed by the image assessment module 204. The method 800 isperformed in connection with generating 606 the assessment information118 of the method 600 and as described elsewhere herein. The method 800begins by identifying 802 facial features in the face image 114 beingevaluated. The system 100 analyzes the face image 114 and identifieswhich, if any, of a set of defined facial features are present in theface image 114. The system 100 may identify regions, e.g., by pixelcoordinates or locations, in the face image 114 that correspond to thefacial features identified. The system 100 may then generate informationidentifying the types of facial feature (e.g., eye, nose, mouth) and theregion corresponding to each facial feature. The information regardingthe facial features may identify locations of particular portions of thefacial features, such as locations of pupil centers and locations ofmouth corners.

Next, the system determines 804 distances between the facial featuresidentified in 802. Determining 804 the distances may involve determiningthe distance d1 between pupil centers in the face image 114 and adistance d2 between corners of the mouth, as shown in FIG. 5. Thedistances d1 and d2 may be distances in directions parallel to ahorizontal axis of the face image 114. That is, the distance d1 may be adistance between a vertical axis extending vertically through the leftpupil center in the face image 114 and a vertical axis extendingvertically through the right pupil center in the face image 114. Thedistance d2 may be a distance between a vertical axis extendingvertically through the left mouth corner in the face image 114 and avertical axis extending vertically through the right mouth corner in theface image 114. Determining 804 distances between facial features mayalso include determining a distance d3 between the eyes and the mouth inthe face image 114. In particular, the distance d3 is determined betweenthe first horizontal line 516 extending between the eye regions and thesecond horizontal line 518 extending between the mouth corner regions.

The method 800 then proceeds to determining 806 dimensions of regionsthat are to be allocated for and relative to the facial features of theface image 114. In particular, the system 100 determines dimensions ofthe regions 502, 504, 506, 508, 510, 512, and 514 (collectively “theregions”). The regions to be determined are rectangular in shape, anddimensions of each of the regions are calculated based on the distancesd1, d2, and d3. Each of the first region 502, the second region 504, thefifth region 510, and the seventh region 514 may have a width (i.e., inthe horizontal direction) equal to half the distance d1—that is a widthof (d1)/2. Each of the third region 506, the fourth region 508, and thesixth region 512 may have a width equal to half the distance d2—that is,a width of (d2)/2. Each of the regions 502, 504, 506, 508, 510, 512, and514 has a height (i.e., in the vertical direction) equal to half thedistance d3—that is, a height of (d3)/2. In some instances, however, thedimensions of some or all of the regions may be different. For example,for a face image 114 in which only a single mouth corner or single pupilis visible, the width of some regions may be based on distances to otherfacial features, such as a bridge of the nose.

At 808 of the method 800, the system allocates 808 the regions in theface image 114 around or relative to facial features. The system 100allocates the first region 502 centered on the left pupil center, thesecond region 504 centered on the right pupil center, the third region506 on the left mouth corner, and the fourth region 508 at the rightmouth corner. The fifth region 510 is allocated to have a verticalcenter axis 520 thereof vertically centered along the first horizontalline 516 with the right edge of the fifth region 510 positioned at or onthe left edge of the first region 502 and the left edge of the fifthregion 510 positioned at or on the right edge of the second region 504.The sixth region 512 is allocated to have a vertical center axis 522thereof vertically centered along the second horizontal line 518 withthe right edge of the sixth region 512 positioned at or on the left edgeof the third region 506 and the left edge of the sixth region 512positioned at or on the right edge of the fourth region 508. The seventhregion 514 is allocated to be positioned between the fifth region 510and the sixth region 512 with an upper edge positioned at or on thelower edge of the fifth region 510 and a lower edge positioned at or onthe upper edge of the sixth region 512. The seventh region 514 isfurther allocated to have a vertical center axis 501 positioned halfway,in the horizontal direction, between the vertical axis 501 of the fifthregion 510 and the vertical axis 501 of the sixth region 512. At theconclusion of the method 800, the system 100 proceeds to determine thecontrast quality information 310 for the regions.

FIG. 9 shows a method 900 for determining the contrast qualityinformation 310 for the regions allocated in the method 800. The method900 may be performed by the image assessment module 204. The method 900is performed in connection with generating 606 the assessmentinformation 118 of the method 600 and as described elsewhere herein. Themethod 900 begins by computing 902 maximum differences between pixelvalues for pixels in each of the regions 502, 504, 506, 508, 510, 512,and 514. The pixel value for a pixel corresponds to an alphanumericvalue representing the brightness of the pixel. The maximum differencebetween pixel values is computed by obtaining, for each region, aminimum pixel value and a maximum pixel value and determining thedifference between the maximum pixel value and the minimum pixel value.

The method 900 proceeds by discarding 904 the maximum pixel differencefor one or more regions. In particular, the system 100 may include adefined number of regions for which to determine the contrast qualityinformation 310. In some embodiments, the defined number of regions maybe five. The system 100 discards 904 the maximum pixel differences forthe regions having the lowest pixel differences to reach the definednumber. In this example, the system 100 discards the lowest two maximumpixel differences to reach five maximum pixel differences. In someinstances, the defined number of maximum pixel differences to discardmay change according to other information. For instance, if the poseinformation 306 indicates that the face in the face image 114 hascertain orientations (e.g., face is looking 45° to one side), the numberof maximum pixel differences to discard may be fewer than five.

The method 900 includes computing 906 an average pixel difference of theremaining regions. That is, the system 100 computes an average maximumpixel difference for the regions 502, 504, 506, 508, 510, 512, and 514that were not discarded in 904. Computing 906 the average pixeldifference may include computing a sum of the maximum pixel differencesfor the regions 502, 504, 506, 508, 510, 512, and 514 that were notdiscarded in 904.

The method 900 may then proceed to normalizing 908 the average maximumpixel difference computed in 906. Normalizing 908 may involvedetermining into which of a plurality of ranges the average maximumpixel difference falls, and calculating a corresponding normalized valuebased on the range determined. FIG. 13 shows a graph representing rangesfor the calculated average maximum pixel difference values andcorresponding normalized values on a 0.0 to 1.0 scale. The system 100determines which range the calculated value (CV) falls into from lessthan T1, T1 to T2, T2 to T3, or greater than T3. Subsequent todetermining into which range the calculated value falls, the system maycalculate a corresponding normalized value (NV) using a formula for therange. The formula for each range may be as follows:If CV<T1,then NV=CV/(T1×0.2);If T1≤CV<T2,then NV=0.2+(CV−T1)/(T2−T1)×0.4;If CV≥T2,then NV=0.6+(CV−T2)/(T3−T2)×0.4

The resulting normalized value (NV) is a unitless value from 0.0 to 1.0.The normalized average maximum pixel difference value and/or thecalculated average maximum pixel difference value may be included in theassessment information 118. In some embodiments, the system 100 maydetermine a category or classification for the calculated averagemaximum pixel difference value instead of determining a normalizedaverage maximum pixel difference value. For example, the system 100 mayclassify a calculated average maximum pixel difference value between T1and T2 as being sufficient to recognize an identity of a person in aface image 114 and classify a calculated average maximum pixeldifference value greater than T2 as being sufficient to learn or updatean identity of the person in the face image 114. As a result ofnormalizing 908 the calculated value, the method 900 includes providing910 contrast quality information 310 corresponding to the average pixeldifference value computed.

FIG. 10 shows a method 1000 for performing facial recognition processingaccording to one or more embodiments. The method 1000 may be performedby one or more appropriate components of the system 100, such as theimage processing module 210. The method 1000 is performed in connectionwith advancing 610 the face image 114 for facial recognition processingof the method 600 and as described elsewhere herein. The method 1000begins by receiving 1002 or otherwise obtaining the assessment output206 indicating result information 208 regarding the assessmentperformed. The result information 208 may include the assessmentinformation 118 or otherwise indicate that certain facial recognitionprocessing operations are to be performed. Receiving 1002 the resultinformation 208 may include receiving the face image 114 or a referencethereto.

The method 1000 includes determining 1004 whether to update an existingidentity or learn a new identity based on the result information 208.The result information 208 may indicate that the image assessmentinformation 118 satisfies criteria for learning or updating an identityaccording to the face image criteria 108. In some embodiments, theresult information 208 specifies which of the face image criteria 108the assessment information 118 satisfies and the system 100 maydetermine whether to update or learn an identity based on the criteriaspecified. In some embodiments, the result information 208 may includeinstructions instructing to update an identity or learn a new identity.In some embodiments, the result information 208 may include some or allof the assessment information 118, which is used to determine whether toupdate or learn an identity based on the face image criteria 108. If itis determined to update or learn the identity in 1004, the method 1000proceeds to update 1006 the data storage 214 as described with respectto FIG. 11 below and elsewhere herein.

If the system 100 determines 1004 not to update or learn an identity,e.g., by virtue of the assessment information 118 failing to satisfy anappropriate set of criteria, the system 100 proceeds to determine 1008whether to identify the person depicted in the face image 114 based onthe result information 208. The result information 208 may indicate thatthe image assessment information 118 satisfies criteria identifying aperson according to the face image criteria 108. In some embodiments,the result information 208 specifies which of the face image criteria108 the assessment information 118 satisfies and the system 100 maydetermine to identify the person depicted in the face image 114 based onthe criteria specified. In some embodiments, the result information 208may include instructions instructing the system 100 to identify theperson depicted in the face image 114. In some embodiments, the resultinformation 208 may include some or all of the assessment information118, which is used to determine 1008 whether to identify the personbased on the face image criteria 108.

If it is determined in 1008 to identify the person depicted in the faceimage 114, the method 1000 proceeds to generate 1010 a biometricsignature 212 of the face image 114. Then, the system 100 compares 1012the biometric signature 212 generated to the stored biometric signatures216 in the data storage 214 to determine a match. If the biometricsignature 212 generated is sufficiently similar to one of the storedbiometric signatures 216, the system 100 returns information indicatingthe identity of the person depicted in the face image 114. The processof comparing 1012 the biometric signature 212 to the stored biometricsignatures 216 is potentially time and resource intensive. By comparingthose face images 114 having attributes matching the face image criteria108, the rate of false positives is reduced and the efficiency of thesystem 100 is improved. That is, the resources of the system 100 aredirected to performing facial recognition processes on select faceimages 114 having high resolution, sharpness, and contrast quality, andthose representing face poses likely to produce a positiveidentification. If, in 1008, the system 100 determines not to identify1008 the person in the face image 114, the face image 114 is discarded1014 (e.g., not stored in memory), and the system 100 proceeds to obtain612 the next face image 114 in the method 600 or proceeds to another setof operations defined herein.

FIG. 11 shows a method 1100 of updating data storage according to one ormore embodiments. The method 1100 is performed as part of or inconnection with updating 1006 data storage in the method 1000. Themethod 1100 may be performed by one or more appropriate components ofthe system 100, such as the image processing module 210. The method 1100is performed as a result of the system 100 determining that the faceimage 114 has characteristics sufficient to satisfy a set of criteriafor learning, updating, or merging identities. The set of criteria forperforming the method 1100 may be different than the criteria fordetermining whether to identify a person in a face image 114. It may bedesirable to use high quality face images 114 as a basis for determiningwhich face images 114 are suitable to use as a reference for identifyingpersons, so the set of criteria for performing the method 1100 may behigher than the set of criteria for determining whether to identify aperson in 1008 of FIG. 10.

The method 1100 begins by generating 1102 a biometric signature 212 ofthe face image 114. Then, the biometric signature 212 generated iscompared 1102 with the stored biometric signatures 216 to determine amatch. If the system 100 determines 1104 that there is no match betweenthe biometric signature 212 and one of the stored biometric signatures216 (No at 1106), the method 1100 may proceed to generate 1108 a newidentity for the biometric signature 212 generated in 1102. This mayinclude obtaining identifying information regarding the person depicted,such as name, age, identification numbers. In some embodiments, theinformation regarding the identity of the person may be obtained viauser input. Thereafter, the method 1100 may terminate and the system 100may return to another operation described herein, such as obtaining 612the next face image 114 to be evaluated in the method 600.

If the system 100 determines 1104 that a match exists between thebiometric signature 212 and one of the stored biometric signatures 216(Yes at 1106), the method 1100 next proceeds to determine whether thebiometric signature 212 is a match to multiple identities. For instance,the system 100 may determine that the biometric signature 212 issufficiently similar to a plurality of stored biometric signatures 216each corresponding to a different identity or person. In such an event,the method 1100 may proceed to merge 1112 the plurality of identities.Merging 1112 the identities may include evaluating the assessmentinformation 118 associated with the face image 114 of the biometricsignature 212 under consideration. The system 100 may determine whetherthe assessment information 118 satisfies a set of criteria for mergingidentities as a condition precedent to merging 1112. The set of criteriafor merging 1112 identities may include different criteria than the setof criteria for updating 1004 or identifying 1008 discussed with respectto the method 1000—for example, the set of criteria for merging 1112 mayinclude higher thresholds for the size information 304, the poseinformation 306, the sharpness information 308, or the contrast qualityinformation 310. In some embodiments, the system 100 may not merge 1112the identities if information stored in association with the storedbiometric signatures 216 if there are inconsistencies associated withthe identities. If, for example, the identities have different socialsecurity numbers associated or the there is an indication that a twinexists for the person, the system 100 may not merge the identities.

Next, the method 1100 proceeds by determining 1114 whether theassessment information 112 represents an improvement over the assessmentinformation 118 of the stored biometric signature 216 matched with thebiometric signature 212. If so, the method 1100 proceeds to replace 1116the corresponding stored biometric signature 216 with the currentbiometric signature 212 and the associated assessment information 118.This will ensure that the biometric signature 212 used as a basis fordetermining the identity of a person is of high quality.

At 1118, the method 1100 involves determining whether the storedbiometric signature 216 for which a match was determined exceeds adefined time threshold. In particular, the system 100 determines a timeperiod between a current time and date and the time and date at whichthe stored biometric signature 216 was captured. If the time periodexceeds a defined time period threshold, the stored biometric signature216 is replaced 1116 with the current biometric signature 212. Thedefined time period threshold is an alphanumeric value stored in memorythat indicates a time period after which the stored biometric signature216 should be replaced. This ensures that the stored biometricsignatures 216 are recent so that old face images 114 are not used as abasis for facial recognition processing. If the time period for thestored biometric signature 216 does not exceed the time periodthreshold, the method 1100 proceeds to provide 1120 identity informationregarding the identity of the person with whom the biometric signature212 was matched in 1106.

FIG. 12 shows a method 1200 for verifying the consistency of the images104 being evaluated or subimages thereof. The method 1200 is performedas part of or in connection with 607 of the method 600. The method 1200may be performed by one or more appropriate components of the system100, such as the image assessment module 204. The method 1200 begins byreceiving 1202 a first image captured by the camera 106. In someembodiments, the first image may be a first image 104 of the pluralityof images 105 that contains one or more face images 114 to be evaluated.In some embodiments, the first image may be the face image 114 currentlybeing evaluated in an image 104 of the plurality of images 105. Next,the method 1200 includes generating 1204 a first signature of the firstimage. For instance, the system 100 may generate a biometric signature212 of the face image 114 being evaluated.

The method 1200 includes receiving 1206 a second image captured by thecamera 106. In some embodiments, the second image is an image 104sequential to the first image in the plurality of images 105. In someembodiments, the second image is the face image 114 of the same personcaptured in an image 104 sequential to the first image in the pluralityof images 105. Thereafter, the method 1200 includes generating 1208 asecond signature of the second image received in 1206—for example,generating a biometric signature of the face image 114 corresponding tothe second image.

The method 1200 further includes comparing 1210 the first signature andthe second signature to determine a degree of similarity orcorrespondence therebetween. For instance, the system 100 may comparevectors and points between the first and second signatures to determinehow similar they are. The system 100 may generate a set of numericalvalues indicating differences between the first and second signatures.At 1212, the method 1200 involves determining whether the first andsecond images are consistent based on the comparison of the signaturesin 1210. The determination 1212 of whether the first image and thesecond image are consistent may be based on a comparison to a thresholdor range for acceptable consistency or similarity.

If the system 100 determines that a result of the comparison 1210indicates sufficient similarity between the first image and the secondimage, the method 1200 proceeds to authorize 1214 facial recognitionprocessing. For instance, the system 100 may generate result information208 indicating that the image being evaluated is consistent or stable.If, on the other hand, it is determined that the result of thecomparison 1210 indicates insufficient similarity, the method 1200proceeds to deny 1216 facial recognition processing for the image 104 orface image 114 being considered for evaluation.

Consistency, as discussed herein, indicates that sequential images 104are stable such that the image 104 or face image 114 being evaluated isundistorted or not blurred. Inconsistent images 104 or face images 114may capture subjects who are in motion or in which other subject mattermay disrupt image evaluation. For instance, cigarette smoke in an imagemay distort a face image 114 captured such that it has difficultfeatures to discern. As another example, a person running or in theprocess of turning may have blurred features if captured during motion.Verifying the consistency of the images helps to ensure that resourcesof the system 100 are not wasted on processing blurred, moving, orotherwise suboptimal images.

The various embodiments described above can be combined to providefurther embodiments. All of the U.S. patents, U.S. patent applicationpublications, U.S. patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet areincorporated herein by reference, in their entirety. Aspects of theembodiments can be modified, if necessary to employ concepts of thevarious patents, applications and publications to provide yet furtherembodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

The invention claimed is:
 1. A computer-implemented method, comprising:receiving a plurality of images; and for each of one or more face imagesin the plurality of images: generating assessment information thatincludes contrast information regarding a contrast quality of the faceimage; comparing the assessment information for the face image with afirst set of criteria that includes a first contrast quality criterion;determining, as a result of comparing the assessment information withthe first set of criteria, whether the assessment information associatedwith the face image satisfies the first set of criteria; identifying, inresponse to determining that the assessment information satisfies thefirst set of criteria, the face image as being qualified for facialrecognition processing; determining that the assessment informationsatisfies a second set of criteria that includes a second contrastquality criterion; generating, as a result of determining that theassessment information satisfies the second set of criteria, a biometricsignature of the face image; and storing the biometric signature as aresult of determining that the assessment information represents animprovement over a stored biometric image.
 2. The computer-implementedmethod of claim 1, wherein the assessment information includes sharpnessinformation regarding a sharpness of the face image, the set of criteriaincludes a sharpness criterion, and determining that the assessmentinformation satisfies the first set of criteria includes determiningthat the sharpness information satisfies the sharpness criterion.
 3. Thecomputer-implemented method of claim 1, wherein generating theassessment information includes: identifying a plurality of features inthe face image that correspond to predetermined facial features;determining a plurality of regions of the face image that respectivelysurround a feature of the plurality of features; assigning contrastvalues to each region of the plurality of regions based on a maximumpixel contrast of pixels in each region; and determining a sum of thecontrast values of the plurality of regions.
 4. The computer-implementedmethod of claim 3, wherein determining that the assessment informationsatisfies the first set of criteria includes determining that the sum ofthe contrast values satisfies the contrast criterion.
 5. Thecomputer-implemented method of claim 1, wherein the first set ofcriteria comprises an orientation criterion, and generating theassessment information includes generating orientation information thatincludes estimating an orientation of the face in the face image, anddetermining that the assessment information satisfies the first set ofcriteria includes determining that the orientation information satisfiesthe orientation criterion.
 6. The computer-implemented method of claim1, wherein the assessment information includes size informationidentifying one or more size dimensions of the face image, the first setof criteria includes a size criterion, and determining that theassessment information satisfies the first set of criteria includesdetermining that the size information satisfies a size threshold in thefirst set of criteria.
 7. The computer-implemented method of claim 1,wherein the face image is a first face image, the assessment informationis first assessment information including first contrast information,the biometric signature is a first biometric signature, and the methodfurther comprising: obtaining a second face image; generating secondassessment information of the second face image that includes secondcontrast information regarding a contrast quality of the second faceimage; determining that the second assessment information satisfies thesecond set of criteria that includes the second contrast qualitycriterion; and generating, as a result of determining that the secondassessment information satisfies the second set of criteria, a secondbiometric signature of the second face image.
 8. Thecomputer-implemented method of claim 7, further including comparing thefirst biometric signature of the first face image to the secondbiometric signature of the second face image.
 9. Thecomputer-implemented method of claim 1, wherein each criterion of thesecond set of criteria includes different values than each correspondingcriterion of the first set of criteria.
 10. The computer-implementedmethod of claim 7, wherein the second face image is obtained at a secondtime, the method further comprising: storing the first biometricsignature of the first face image as being associated with an identityin data storage at a first time previous to the second time; and storingthe second biometric signature of the second face image as a result of adetermination that a time period between the first time and the secondtime exceeds a predetermined threshold.
 11. The computer-implementedmethod of claim 1, further comprising: generating a second biometricsignature for a second face image consecutive to the face image in theplurality of images; and determining an image differential based on acomparison of the biometric signature and the second biometricsignature, wherein analyzing the face in the face image is performed asa result of the image differential being less than a predeterminedthreshold for image consistency.
 12. The method of claim 1, furthercomprising: detecting face images in at least a subset of images of theplurality of images, wherein generating the assessment information is inresponse to detecting the face image in the subset of images.
 13. Themethod of claim 1, further comprising: analyzing the face in the faceimage to determine an identity of a person corresponding to the face.14. A system, comprising: one or more processors; and non-transitorystorage medium storing a set of instructions that, as a result ofexecution by the one or more processors, cause the system to: receive aplurality of images; and for each of one or more face images in theplurality of images: generate assessment information that includescontrast information regarding a contrast quality of the face image;compare the assessment information with a first set of criteria thatincludes a first contrast quality criterion; generate, as a result ofdetermining that the assessment information satisfies the first set ofcriteria, result information indicating that the assessment informationassociated with the face image satisfies the first set of criteria;compare the assessment information with a second set of criteria thatincludes a second contrast quality criterion; generate, as a result ofdetermining that the assessment information satisfies the second set ofcriteria, a biometric signature of the face image; and store thebiometric signature as a result of determining that the assessmentinformation represents an improvement over a stored biometric image. 15.The system of claim 14, wherein the instructions, as a result ofexecution by the one or more processors, further cause the system to:transmit, to a second system, the result information for authorizing thesecond system to perform facial recognition processing using the faceimage.
 16. The system of claim 14, wherein the instructions, as a resultof execution by the one or more processors, further cause the system to:compare, as a result of determining that the assessment informationsatisfies the first set of criteria, the biometric signature with aplurality of biometric signatures stored in data storage; determine, asa result of comparing the biometric signature with the plurality ofbiometric signatures, that the biometric signature does not correspondto a stored identity; and create a new identity corresponding to thebiometric signature.
 17. The system of claim 14, wherein theinstructions, as a result of execution by the one or more processors,further cause the system to: compare, as a result of determining thatthe assessment information satisfies the first set of criteria, thebiometric signature with a plurality of biometric signatures stored indata storage; determine, as a result of comparing the biometricsignature with the plurality of biometric signatures, that the biometricsignature corresponds to a plurality of stored identities; and merge theplurality of identities into a single stored identity.
 18. The system ofclaim 14, wherein the assessment information includes a size of the faceimage, pose information regarding an orientation of a face in the faceimage, and sharpness information regarding a sharpness of the faceimage, and the first set of criteria includes a size criterion, anorientation criterion, and a sharpness criterion.
 19. The system ofclaim 14, wherein the instructions, as a result of execution by the oneor more processors, further cause the system to: identify a plurality offeatures in the face image that correspond to predetermined facialfeatures; determine a plurality of regions of the face image thatrespectively surround each feature of the plurality of features; computemaximum pixel difference values for the plurality of regions; andcompute an average pixel difference of the maximum pixel differencevalues, wherein the contrast information corresponds to the averagepixel difference.
 20. The system of claim 14, wherein the second set ofcriteria is different from the first set of criteria.
 21. Anon-transitory computer-readable medium storing computer-executableinstructions that, as a result of execution by a processor, cause theprocessor to: obtain a plurality of images; and for each of one or moreface images in the plurality of images: generate assessment informationthat includes contrast information regarding a contrast quality of theface image; compare the assessment information with a first set ofcriteria that includes a first contrast quality criterion; generate, asa result of determining that the assessment information satisfies thefirst set of criteria, result information indicating that the assessmentinformation associated with the face image satisfies the first set ofcriteria; compare the assessment information with a second set ofcriteria that includes a second contrast quality criterion; generate, asa result of determining that the assessment information satisfies thesecond set of criteria, a biometric signature of the face image; andstore the biometric signature as a result of determining that theassessment information represents an improvement over a stored biometricimage.
 22. The non-transitory computer-readable medium of claim 21,wherein the second set of criteria is different from the first set ofcriteria.
 23. A computer-implemented method, comprising: obtaining afirst face image at a first time; generating first assessmentinformation of the first face image that includes first contrastinformation regarding a first contrast quality of the first face image;comparing the first assessment information for the first face image witha first set of criteria that includes a first contrast qualitycriterion; determining, as a result of comparing the first assessmentinformation with the first set of criteria, whether the first assessmentinformation associated with the first face image satisfies the first setof criteria; generating, as a result of determining that the firstassessment information satisfies the first set of criteria, a firstbiometric signature of the first face image storing the first biometricsignature of the first face image as being associated with an identify;obtaining a second face image at a second time; generating secondassessment information of the second face image that includes secondcontrast information regarding a second contrast quality of the secondface image; determining that the second assessment information satisfiesa second set of criteria that includes a second contrast qualitycriterion; generating, as a result of determining that the secondassessment information satisfies the second set of criteria, a secondbiometric signature of the second face image; and storing the secondbiometric signature of the second face image as a result of adetermination that a time period between the first time and the secondtime exceeds a predetermined threshold.